Resolution
Digital cameras are useful in both consumer and professional applications. Generally, digital cameras can acquire a digital image of an object or a scene. The acquired image can then be transferred to a computer system for rendering, inserted as a graphical object into an electronic document, stored in a storage device, or output to a printing device, among other actions. In addition, there are other means of generating a digital image, e.g., by scanning photographs.
A variety of camera capabilities are available. One important camera feature that can significantly impact the cost of a digital camera is its resolution, which is typically measured in pixels. Camera resolutions can range from thousands to millions of pixels. The image resolution determines the level of detail in an image. Thus, higher resolution cameras typically enhance the appearance of images. A higher resolution also generally translates into higher cost. Nevertheless, many users wish to find a camera or other digital image capture device that provides enhanced appearance at a low price.
Motion Blur
FIG. 1C shows the effect of motion blur to be removed according to the embodiments of the invention. Motion blur is the apparent streaking: due to a moving object in an image. For example, if a static object in an image is represented by 10×20 pixels, and the object is moving horizontally, the moving object in the blurred image 192 might become 10×40 pixels, with the increase being in a direction of the motion. Motion blur can degrade the quality of the image, and is considered a nuisance for computer vision applications. Motion blur is the result of relative motion between the camera and the scene during an integration or ‘exposure time’ while acquiring an image. Motion blurred images can be restored up to lost spatial frequencies by image deconvolution, provided that the motion is shift-invariant, at least locally, and that a blur function, also known as a point spread function (PSF), which caused, the blur is known. That is, conventional deblurring can, at best, recover a relatively reasonable appearing 10×20 object.
It is an object of the invention to recover an enhanced appearing object at an increased resolution, e.g., 10×40 pixels.
Several methods are known for deblurring and resolution enhancement.
Super Resolution (SR)
Super resolution (SR) refers to an image processing method for increasing a resolution of an image acquired by a low-resolution camera sensor. Typically, super resolution operates on entire images, and not on specific objects in the images. Numerous methods, which combine multiple low-resolution images into a single high resolution image have been described in the prior art, e.g., H. Ur and D. Gross: “Improved resolution from sub-pixel shifted pictures,” CVGIP: Graphical Models and Image Processing, volume 54, pages 181-186, 1992.
In those methods, the relative motion between the camera and a scene is estimated. Then, all images are registered to a reference image. The images are combined to obtain a high resolution image. However, the motion blur in individual images significantly degrades the quality of the super-resolution. It is a goal of the invention to increase the resolution of blurred objects without degrading the quality of the image.
Another method uses a jitter camera. There, the sensor was modified to obtain four images using controlled sub-pixel detector shifts, see M. Ben-Ezra, A. Zomet, and S. Nayar: “Video super-resolution using controlled subpixel detector shifts,” IEEE Trans. Pattern Anal. Machine Intell., 27:977-987, June 2005.
High resolution recovery of 1-D features was described by F. Champagnat, G. Le Besnerais, and C. Kulesar: “Continuous super-resolution for recovery of 1-D image features: Algorithm and performance modeling,” Proc. Conf. Computer Vision and Pattern Recognition, volume 1, pages 916-926, 2006. However, their method is restricted because of an assumption that each row or column of pixels in the low resolution image results from the sampling of the same signal with different shifts.
Single image SR is an under-constrained problem. Most of the previous work can be classified into: (a) reconstruction based methods, where a high resolution image is desired so that after down-sampling, the high resolution image is as close as possible to the low resolution input image, (b) learning based Bayesian methods using training dataset and image priors, and (c) functional interpolation, which results in blurring of discontinuities in the image.
Neighbor embedding and tensor voting have also been described for single image SR, see H. Chang, D.-Y. Yeung, and Y. Xiong: “Super-resolution through neighbor embedding,” Proc. Conf. Computer Vision and Pattern Recognition, volume 1, pages 275-282, 2004, and Y.-W. Tai, W.-S. Tong, and C.-K. Tang: “Perceptually-inspired and edge-directed color image super-resolution,” Proc. Conf Computer Vision and Pattern Recognition, volume 2, pages 1948-1955, 2006.
Both motion deblurring and resolution enhancement from multiple images are ill-posed problems for images acquired by a conventional camera.
Motion Deblurring
Image deblurring and deconvolution are well-known. Blind image deconvolution attempts to infer concurrently a sharp image and the PSF from a given image, based on various assumptions applied to the PSF.
Bayesian methods assume specific image prior probabilities, such as a Poisson distribution as in the well-known Richardson-Lucy method for astronomical images or natural image statistics for consumer photography.
Ben-Ezra et al. describes a hybrid camera where a low resolution video camera was used to estimate the PSF. The PSF was then used to deblur high resolution image from a digital still camera, see M. Ben-Ezra and S. Nayar: “Motion-based Motion Deblurring,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 26(6); 689-698, June 2004.
Coded Sampling
Methods to preserve spatial frequencies for subsequent reconstruction include coded aperture imaging, frequently used in astronomy to overcome the limitations of a pinhole camera. There, a Modified Uniformly Redundant Arrays (MURA) is used for coding and decoding a light distribution of distant stars using a circular convolution and deconvolution.
Broadband codes are used in spread-spectrum techniques and code-division multiplexing. Broadband sequences are used as training sequences for channel estimation in communication systems over time-dispersive channels.
A coded exposure camera can preserve high spatial frequencies in a motion-blurred image and make the deblurring process well-posed, see U.S. patent application Ser. No. 11/429,694, “Method for Deblurring Images using Optimized Temporal Coding Patterns” filed by Raskar et al. on May 5, 2006, and incorporated hearing by reference.
It is desired to increase the resolution of a moving object in a motion-blurred image.